Feb 23
17
Gary Illyes posted a new blog post on the Google Search Central site asking all of you to stop using 403 and 404 server status codes to reduce the crawl rate of Googlebot. He said they have seen an uptick in the number of sites and CDNs doing this and they need to cut it out.
See the article here:
Google: Stop Using 403s or 404s To Reduce Googlebot Crawl Rates